.TH FENCE_SANLOCK 8 2013-05-02

.SH NAME
fence_sanlock \- fence agent using watchdog and shared storage leases

.SH SYNOPSIS
.B fence_sanlock
[OPTIONS]

.SH DESCRIPTION
fence_sanlock uses the watchdog device to reset nodes, in conjunction with
three daemons: fence_sanlockd, sanlock, and wdmd.

The watchdog device, controlled through /dev/watchdog, is available when a
watchdog kernel module is loaded.  A module should be loaded for the
available hardware.  If no hardware watchdog is available, or no module is
loaded, the "softdog" module will be loaded, which emulates a hardware
watchdog device.

Shared storage must be configured for sanlock to use from all hosts.  This
is generally an lvm lv (non-clustered), but could be another block device,
or NFS file.  The storage should be 1GB of fully allocated space.  After
being created, the storage must be initialized with the command:
.br
# fence_sanlock -o sanlock_init -p /path/to/storage

The fence_sanlock agent uses sanlock leases on shared storage to verify
that hosts have been reset, and to notify fenced nodes that are still
running, that they should be reset.

The fence_sanlockd init script starts the wdmd, sanlock and fence_sanlockd
daemons before the cluster or fencing systems are started (e.g. cman,
corosync and fenced).  The fence_sanlockd daemon is started with the -w
option so it waits for the path and host_id options to be provided when
they are available.

Unfencing must be configured for fence_sanlock in cluster.conf.  The cman
init script does unfencing by running fence_node -U, which in turn runs
fence_sanlock with the "on" action and local path and host_id values taken
from cluster.conf.  fence_sanlock in turn passes the path and host_id
values to the waiting fence_sanlockd daemon.  With these values,
fence_sanlockd joins the sanlock lockspace and acquires a resource lease
for the local host.  It can take several minutes to complete these
unfencing steps.

Once unfencing is complete, the node is a member of the sanlock lockspace
named "fence" and the node's fence_sanlockd process holds a resource lease
named "hN", where N is the node's host_id.  (To verify this, run the
commands "sanlock client status" and "sanlock client host_status", which
show state from the sanlock daemon, or "sanlock direct dump <path>" which
shows state from shared storage.)

When fence_sanlock fences a node, it tries to acquire that node's resource
lease.  sanlock will not grant the lease until the owner (the node being
fenced) has been reset by its watchdog device.  The time it takes to
acquire the lease is 140 seconds from the victim's last lockspace renewal
timestamp on the shared storage.  Once acquired, the victim's lease is
released, and fencing completes successfully.

Live nodes being fenced

When a live node is being fenced, fence_sanlock will continually fail to
acquire the victim's lease, because the victim continues to renew its
lockspace membership on storage, and the fencing node sees it is alive.
This is by design.  As long as the victim is alive, it must continue to
renew its lockspace membership on storage.  The victim must not allow the
remote fence_sanlock to acquire its lease and consider it fenced while it
is still alive.

At the same time, a victim knows that when it is being fenced, it should
be reset to avoid blocking recovery of the rest of the cluster.  To
communicate this, fence_sanlock makes a "request" on storage for the
victim's resource lease.  On the victim, fence_sanlockd, which holds the
resource lease, is configured to receive SIGUSR1 from sanlock if anyone
requests its lease.  Upon receiving the signal, fence_sanlockd knows that
it is a fencing victim.  In response to this, fence_sanlockd allows its
wdmd connection to expire, which in turn causes the watchdog device to
fire, resetting the node.

The watchdog reset will obviously have the effect of stopping the victim's
lockspace membership renewals.  Once the renewals stop, fence_sanlock will
finally be able to acquire the victim's lease after waiting a fixed time
from the final lockspace renewal.

Loss of shared storage

If access to shared storage with sanlock leases is lost for 80 seconds,
sanlock is not able to renew the lockspace membership, and enters
recovery.  This causes sanlock clients holding leases, such as
fence_sanlockd, to be notified that their leases are being lost.  In
response, fence_sanlockd must reset the node, much as if it was being
fenced.

Daemons killed/crashed/hung

If sanlock, fence_sanlockd daemons are killed abnormally, or crash or
hang, their wdmd connections will expire, causing the watchdog device to
fire, resetting the node.  fence_sanlock from another node will then run
and acquire the victim's resource lease.  If the wdmd daemon is killed
abnormally or crashes or hangs, it will not pet the watchdog device,
causing it to fire and reset the node.

Time Values

The specific times periods referenced above, e.g. 140, 80, are based on
the default sanlock i/o timeout of 10 seconds.  If sanlock is configured
to use a different i/o timeout, these numbers will be different.

.SH OPTIONS

.BI \-o " action"
    The agent action:

.IP
.B on
.br
Enable the local node to be fenced.  Used by unfencing.

.IP
.B off
.br
Disable another node.

.IP
.B status
.br
Test if a node is on or off.  A node is on if it's lease is held, and off
is it's lease is free.

.IP
.B metadata
.br
Print xml description of required parameters.

.IP
.B sanlock_init
.br
Initialize sanlock leases on shared storage.

.PP

.BI \-p " path"
    The path to shared storage with sanlock leases.

.PP

.BI \-i " host_id"
    The host_id, from 1-128.

.SH STDIN PARAMETERS

Options can be passed on stdin, with the format key=val.  Each key=val
pair is separated by a new line.

action=on|off|status
.br
See \-o

path=/path/to/shared/storage
.br
See \-p

host_id=num
.br
See \-i

.SH FILES 

Example cluster.conf configuration for fence_sanlock.
.br
(For cman based clusters in which fenced runs agents.)
.br
Also see cluster.conf(5), fenced(8), fence_node(8).

.nf
<clusternode name="node01" nodeid="1">
        <fence>
        <method name="1">
        <device name="wd" host_id="1"/>
        </method>
        </fence>
        <unfence>
        <device name="wd" host_id="1" action="on"/>
        </unfence>
</clusternode>

<clusternode name="node02" nodeid="2">
        <fence>
        <method name="1">
        <device name="wd" host_id="2"/>
        </method>
        </fence>
        <unfence>
        <device name="wd" host_id="2" action="on"/>
        </unfence>
</clusternode>

<fencedevice name="wd" agent="fence_sanlock" path="/dev/fence/leases"/>
.fi

.PP

Example dlm.conf configuration for fence_sanlock.
.br
(For non-cman based clusters in which dlm_controld runs agents.)
.br
Also see dlm.conf(5), dlm_controld(8).

.nf
device wd /usr/sbin/fence_sanlock path=/dev/fence/leases
connect wd node=1 host_id=1
connect wd node=2 host_id=2
unfence wd
.fi

.SH TEST

To test fence_sanlock directly, without clustering:

.nf
1. Initialize storage

node1: create 1G lv on shared storage /dev/fence/leases
node1: fence_sanlock -o sanlock_init -p /dev/fence/leases

2. Start services

node1: service fence_sanlockd start
node2: service fence_sanlockd start

3. Enable fencing

node1: fence_sanlock -o on -p /dev/fence/leases -i 1
node2: fence_sanlock -o on -p /dev/fence/leases -i 2

This "unfence" step may take a couple minutes.

4. Verify hosts and leases

node1: sanlock status
s fence:1:/dev/fence/leases:0
r fence:h1:/dev/fence/leases:1048576:1 p 2465

node2: sanlock status
s fence:2:/dev/fence/leases:0
r fence:h2:/dev/fence/leases:2097152:1 p 2366

node1: sanlock host_status
lockspace fence
1 timestamp 717
2 timestamp 678

node2: sanlock host_status
lockspace fence
1 timestamp 738
2 timestamp 678

5. Fence node2

node1: fence_sanlock -o off -p /dev/fence/leases -i 2

This may take a few minutes to return.

When node2 is not dead before fencing, sanlock on node1 will log errors
about failing to acquire the lease while node2 is still alive.  This is
expected.

6. Success

node1 fence_sanlock should exit 0 after node2 is reset by its watchdog.
.fi


.SH SEE ALSO
.BR fence_sanlockd (8),
.BR sanlock (8),
.BR wdmd (8)

